Self-Modeling Agents and Reward Generator Corruption

Hutter's universal artificial intelligence (AI) showed how to define future AI systems by mathematical equations. Here we adapt those equations to define a self-modeling framework, where AI systems learn models of their own calculations of future values. Hutter discussed the possibility that AI agents may maximize rewards by corrupting the source of rewards in the environment. Here we propose a way to avoid such corruption in the selfmodeling framework. This paper fits in the context of my book Ethical Artificial Intelligence. A draft of the book is available at: arxiv.org/abs/1411.1373. Self-Modeling Agents Russell and Norvig defined a framework for AI agents interacting with an environment (Russell and Norvig 2010). Hutter adapted Solomonoff's theory of sequence prediction to this framework to produce mathematical equations that define behaviors of future AI systems (Hutter 2005). Assume that an agent interacts with its environment in a discrete, finite series of time steps t ∈ {0, 1, 2, ..., T}. The agent sends an action at ∈ A to the environment and receives an observation ot ∈ O from the environment, where A and O are finite sets. We use h = (a1, o1, ..., at, ot) to denote an interaction history where the environment produces observation oi in response to action ai for 1 ≤ i ≤ t. Let H be the set of all finite histories so that h ∈ H, and define |h| = t as the length of the history h. An agent's predictions of its observations are uncertain so the agent's environment model takes the form of a probability distribution over interaction histories: